Face model fitting can allow a range of human to computer interaction applications. Examples of such applications include, but are not limited to, live facial puppetry, facial feature detection, head pose estimation, face tracking, face recognition, facial expression recognition, and any combinations thereof. Prior techniques for face model fitting have several shortcomings, especially when run on lower powered computing devices, such as a mobile device running an ARM processor. Examples of shortcomings of prior techniques include, but are not limited to, intensive training stages, requirements for initial head pose estimation, difficulty tracking 3D movement, difficulty in variations from one person to another, and others.